Driver assistance system for a vehicle

ABSTRACT

A driver assistance system for a vehicle includes a first imager disposed at a left side of the vehicle, a second imager disposed at a rear of the vehicle and a third imager disposed at a right side of the vehicle. The first, second and third imagers have respective fields of view and the first imager is spatially separated from the second imager and the third imager is spatially separated from the second imager. A display system has a single display screen that is viewable by a driver of the vehicle. The display screen is operable to display an image derived from image data captured by the first, second and third imagers. The display screen displays an image synthesized from image data captured by each of the first, second and third imagers. The displayed image approximates a view from a single virtual camera location.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 11/649,064, filed Jan. 3, 2007, which is a continuation of U.S.patent application Ser. No. 10/643,602, filed Aug. 19, 2003, now U.S.Pat. No. 7,859,565, which is a continuation of U.S. patent applicationSer. No. 09/776,625, filed Feb. 5, 2001, now U.S. Pat. No. 6,611,202,which is a continuation of U.S. patent application Ser. No. 09/313,139,filed May 17, 1999, now U.S. Pat. No. 6,222,447, which is a continuationof U.S. patent application Ser. No. 08/935,336, filed Sep. 22, 1997, nowU.S. Pat. No. 5,949,331, which is a continuation of U.S. patentapplication Ser. No. 08/445,527, filed May 22, 1995, now U.S. Pat. No.5,670,935.

BACKGROUND OF THE INVENTION

This invention relates generally to vision systems for vehicles and,more particularly, to rearview vision systems which provide the vehicleoperator with scenic information in the direction rearward of thevehicle. More particularly, the invention relates to a rearview visionsystem utilizing image capture devices, such as CMOS imaging arrays andthe like.

A long-felt need in the art of vehicle rearview vision systems has beento eliminate exterior rearview mirrors by utilizing image capturedevices, such as cameras, in combination with dashboard displays. Thiswould be beneficial because it would reduce wind drag on the vehicle,wind noise and vehicle weight. Furthermore, rearview mirrors protrude asubstantial distance from the side of the vehicle, which makesmaneuvering in tight spaces more difficult. Image capture devices arecapable of positioning in a greater variety of locations on the vehicle,providing more flexibility of vehicle styling. It is further expectedthat camera systems would greatly reduce the blind spots to the sidesand rear of the vehicle common with vehicles equipped with conventionalrearview mirror systems. The driver cannot perceive vehicles, objects,or other road users in such blind spots without turning his or her body,which interferes with forward-looking visual activities.

Camera-based rearview vision systems for vehicles have not obtainedcommercial acceptance. One difficulty with proposed systems has beenthat they present a large amount of visual information in a manner whichis difficult to comprehend. This difficulty arises from many factors. Inorder to significantly reduce blind spots, multiple image capturedevices are typically positioned at various locations on the vehicle.The image of an object behind the equipped vehicle is usually capturedby more than one image capture device at a time and displayed inmultiple images. This may confuse the driver as to whether more than oneobject is present. When multiple image capture devices are positioned atdifferent longitudinal locations on the vehicle, objects behind thevehicle are at different distances from the image capture devices. Thisresults in different image sizes for the same object. This effect isespecially noticeable for laterally extending images, such as a bridge,highway crosswalk markings, the earth's horizon, and the like. Suchimages are at different vertical angles with respect to the imagecapture devices. This results in different vertical positions on thedisplay causing the elongated image to appear disjointed.

A camera system provides a monocular view of the scene, compared to thebinocular, or stereoscopic, view obtained when the scene is viewedthrough a rearview mirror. This makes the ability to judge distances ina camera system a problem. This effect is most noticeable at distancesclose to the vehicle where stereoscopic imaging is relied uponextensively by the driver in judging relative locations of objects.Therefore, known camera systems fail to provide to the driver importantinformation where that information is most needed at small separationdistances from surrounding objects.

Another difficulty with camera systems is that, in order to provide asufficient amount of information, the camera system typically presentsthe driver with a greatly increased field of view. This improvesperformance by further reducing blind spots at the side and rear of thevehicle. However, an increased field of view is often obtained byutilizing a wide-angle lens which introduces distortion of the scene andfurther impairs the ability of the driver to judge distances of objectsdisplayed. The problem with such distortion of the scene is that thedriver must concentrate more on the display and take a longer time tointerpret and extract the necessary information. This further distractsthe driver from the primary visual task of maintaining awareness ofvehicles and other objects in the vicinity of the driven vehicle.

SUMMARY OF THE INVENTION

The present invention is directed towards enhancing the interpretationof visual information in a rearview vision system by presentinginformation in a manner which does not require significant concentrationof the driver or present distractions to the driver. This isaccomplished according to the invention in a rearview vision systemhaving at least two image capture devices positioned on the vehicle anddirected rearwardly with respect to the direction of travel of thevehicle. A display is provided for images captured by the image capturedevices. The display combines the captured images into an image thatwould be achieved by a single rearward-looking camera having a viewunobstructed by the vehicle. In order to obtain all of the necessaryinformation of activity, not only behind but also along side of thevehicle, the virtual camera should be positioned forward of the driver.The image synthesized from the multiple image capture devices may have adead space which corresponds with the area occupied by the vehicle. Thisdead space is useable by the driver's sense of perspective in judgingthe location of vehicles behind and along side of the equipped vehicle.

The present invention provides techniques for synthesizing imagescaptured by individual, spatially separated, image capture devices intosuch ideal image, displayed on the display device. This may beaccomplished according to an aspect of the invention by providing atleast three image capture devices. At least two of the image capturedevices are side image capture devices mounted on opposite sides of thevehicle. At least one of the image capture devices is a center imagecapture device mounted laterally between the side image capture devices.A display system displays an image synthesized from outputs of the imagecapture devices. The displayed image includes an image portion from eachof the image capture devices. The image portion from the center imagecapture device is vertically compressed.

It has been discovered that such vertical compression substantiallyeliminates distortion resulting from the spatial separation between thecameras and can be readily accomplished. In an illustrated embodiment,the image compression is carried out by removing selective ones of thescan lines making up the image portion. A greater number of lines areremoved further away from the vertical center of the image.

The compression of the central image portion produces a dead space inthe displayed image which may be made to correspond with the area thatwould be occupied by the vehicle in the view from the single virtualcamera. Preferably, perspective lines are included at lateral edges ofthe dead space which are aligned with the direction of travel of thevehicle and, therefore, appear in parallel with lane markings. Thisprovides visual clues to the driver's sense of perspective in order toassist in judging distances of objects around the vehicle.

According to another aspect of the invention, image enhancement meansare provided for enhancing the displayed image. Such means may be in theform of graphic overlays superimposed on the displayed image. Suchgraphic overlap may include indicia of the anticipated path of travel ofthe vehicle which is useful in assisting the driver in guiding thevehicle in reverse directions. Such graphic overlay may include adistance grid indicating distances behind the vehicle of objectsjuxtaposed with the grid.

These and other objects, advantages, and features of this invention willbecome apparent by review of the following specification in conjunctionwith the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a top plan view of a vehicle having a rearview vision systemaccording to the invention;

FIG. 2 is a side elevation of the vehicle in FIG. 1;

FIG. 3 is a front elevation of a display according to the invention;

FIG. 4 is the same view as FIG. 1 illustrating an alternative embodimentof the invention;

FIG. 5 is a block diagram of an electronic system according to theinvention;

FIG. 6 is the same view as FIG. 3 illustrating an alternate mode ofoperation of the system;

FIG. 7 is the same view as FIG. 2 illustrating an alternative embodimentof the invention;

FIG. 8 is the same view as FIG. 3 illustrating an alternative embodimentof the invention;

FIG. 9 is the same view as FIGS. 1 and 4 illustrating an alternativeembodiment of the invention;

FIG. 10 is the same view as FIGS. 3 and 8 illustrating an alternativeembodiment of the invention; and

FIG. 11 is a chart illustrating the horizontal row of pixels (n1, n2) onwhich an object will be imaged from two longitudinally separated imagecapture devices as that object is spaced at different longitudinaldistances from the image capture devices.

DESCRIPTION OF THE PREFERRED EMBODIMENT

Referring now specifically to the drawings, and the illustrativeembodiments depicted therein, a vehicle 10, which may be an automobile,a light truck, a sport utility vehicle, a van, a bus, a large truck, orthe like includes a rearview vision system, generally illustrated at 12,for providing a driver of the vehicle with a view rearwardly of thevehicle with respect to the direction of travel D of the vehicle (FIG.1). Vision system 12 includes at least two side image capture devices 14positioned, respectively, on opposite sides of vehicle 10 and a centerimage capture device 16 positioned on the lateral centerline of thevehicle. All of the image capture devices are directed generallyrearwardly of the vehicle. Rearview vision system 12 additionallyincludes an image processor 18 for receiving data signals from imagecapture devices 14, 16 and synthesizing, from the data signals, acomposite image 42 which is displayed on a display 20.

As will be set forth in more detail below, the images captured by imagecapture devices 14, 16 are juxtaposed on display 20 by image processor18 in a manner which approximates the view from a single virtual imagecapture device positioned forwardly of the vehicle at a location C andfacing rearwardly of the vehicle, with the vehicle being transparent tothe view of the virtual image capture device. Vision system 12 providesa substantially seamless panoramic view rearwardly of the vehiclewithout duplicate or redundant images of objects. Furthermore,elongated, laterally-extending, objects, such as the earth's horizon,appear uniform and straight across the entire displayed image. Thedisplayed image provides a sense of perspective, which enhances theability of the driver to judge location and speed of adjacent trailingvehicles.

Each of side image capture devices 14 has a field of view 22 and isaimed rearwardly with respect to the vehicle about an axis 24 which isat an angle, with respect to the vehicle, that is half of the horizontalfield of view of the image capture device. In this manner, each of theimage capture devices 14 covers an area bounded by the side of thevehicle and extending outwardly at an angle defined by the horizontalfield of view of the respective side image capture device. Center imagecapture device 16 has a horizontal field of view 26, which issymmetrical about the longitudinal axis of the vehicle. The field ofview of each side image capture device 14 intersect the field of view ofcenter image capture device 16 at a point P which is located a distanceQ behind vehicle 10.

Rear blind zones 30 are located symmetrically behind vehicle 10extending from the rear of the vehicle to point P. Side blind zones 25located laterally on respective sides of the vehicle extend rearwardlyof the forward field of view 36 of the driver to the field of view 22 ofthe respective side image capture device 14. An object will not becaptured by side image capture devices 14 or center image capturedevices 16 if the object is entirely within one of the blind zones 25,30. In order for an object, such as another vehicle V or other road usertravelling to the side of vehicle 10, to be observed by an operator ofvehicle 10, the object must be either at least partially within theforward field of view 36 of the driver or be captured by image capturedevices 14, 16 and displayed on display 20.

FIG. 4 illustrates vehicle 10 travelling on a three-lane highway havinglanes L1, L2, and L3 with the vehicle in lane L2. Another vehicle V isshown positioned mostly within one of the blind zones 25, but with therearmost portion of the vehicle V extending into field of view 22 wherethe vehicle image will be captured by one of side image capture devices14. In the illustrated embodiment, vehicle V is a motorcycle travellingin the center of lanes L1 or L3 and represents a worst case forobserving a vehicle travelling at least partially within one of theblind zones 25. In order for a portion of vehicle V to be extendingeither forwardly or rearwardly of the respective blind zone 25, wherethe vehicle V may be observed by either the forward field of view 36 ofthe driver or by the rearview vision system 12, the field of view 22 ofside image capture devices 14 must be sufficiently wide to capture aportion of vehicle V as illustrated in FIG. 4. Preferably, thehorizontal field of view 22 of side image capture devices 14 is nogreater than that required to provide sufficient coverage which would bein the range of between approximately 55 degrees and approximately 70degrees. In the illustrated embodiment, the horizontal field of view 22is 61 degrees. In order for a portion of vehicle V to be within avertical field of view 40 of one of side image capture devices 14, thefield of view should extend to the pavement at a plane M whichintersects vehicle V (FIG. 2). Preferably, vertical field of view 40 isbetween approximately 60 degrees and approximately 75 degrees. In theillustrated embodiment, vertical field of view 40 is 66 degrees. A leftoverlap zone 32 and a right overlap zone 34 extend rearward fromrespective points P where the horizontal fields of view of the sideimage capture devices intersect the field of view of center imagecapture device 16. Overlap zones 32, 34 define areas within which anobject will be captured both by center image capture device 16 and oneof the side image capture devices 14. An object in an overlap zone 32,34 will appear on display 20 in multiple image portions in a redundantor duplicative fashion. In order to avoid the presentation of redundantinformation to the driver, and thereby avoid confusion and simplify thetask of extracting information from the multiple images or combinedimages on display 20, the object should avoid overlapping zones 32, 34.In practice, this may be accomplished to a satisfactory extent by movingpoints P away from the vehicle and thereby increasing distance Q. It isdesirable to increase distance Q to a length that will exclude vehiclestravelling at a typical separation distance behind vehicle 10 fromoverlapping zones 32, 34. This separation distance is usually a functionof the speed at which the vehicles on the highway are travelling. Thefaster the vehicles are travelling, the further Q should be moved behindvehicle 10 to keep overlap zones 32 and 34 outside of the recommendedvehicle spacing. If, however, the vehicles are travelling at a slowerspeed, then the generally accepted recommendation for vehicle spacingdecreases and it is more likely that a vehicle will be within overlapzone 32, 34. Therefore, the distance Q may be selected to accommodateexpected vehicle spacing for an average driving speed of vehicle 10.

Distance Q is a function of the effective horizontal field of view 26 ofcenter image capture device 16. As field of view 26 decreases, points Pmove further rearward of the vehicle from a distance Q₁, to a distanceQ₂, as best illustrated in FIG. 4. In order to increase distance Q toeliminate redundant and duplicative information displayed on display 20for most driving conditions of vehicle 10, field of view 26 ispreferably less than 12 degrees. In the illustrated embodiment, field ofview 26 is between 6 and 8 degrees. Alternatively, distance Q may bedynamically adjusted according to some parameter, such as the speed ofvehicle 10. This would allow Q to be greater when the vehicle istravelling at a faster speed, where vehicle separation tends to belarger, and vice versa. Field of view 26 may be adjusted by utilizing aselective presentation of pixels of the captured image in the displayedimage.

Referring to FIG. 3, image display device 20 displays a composite image42 made up of a left image portion 44, a right image portion 46, and acenter image portion 48. Each image portion 44-48 is reversed from theimage as captured by the respective image capture device 14, 16utilizing conventional techniques. These techniques include reading theimage in reverse with the image capture device, writing the image inreverse to display device 20, or reversing the image in image processor18. Left image portion 44 is joined with central image portion 48 at aboundary 50. Central image portion 48 is joined with right image portion46 at a boundary 52. As may best be seen in FIG. 3, the image portionsat boundaries 50 and 52 are continuous whereby composite image 42 is aseamless panoramic view rearwardly of the vehicle. As also is apparentfrom FIG. 3, central image portion 48 is narrower than either left imageportion 44 or right image portion 46. This is a result of reducing thehorizontal field of view 26 of center image capture device 16sufficiently to move points P, and thus overlap zones 32 and 34, asufficient distance behind vehicle 10 to reduce redundant andduplicative images between image portions 44-48. Composite image 42provides a clear image, which avoids confusion and simplifies the taskof extracting information from the multiple image portions 44-48. Asalso may be seen by reference to FIG. 3, display 20 may additionallyinclude indicia such as the readout of a compass 54, vehicle speed 56,turn signals 58, and the like as well as other graphical or videodisplays, such as a navigation display, a map display, and aforward-facing vision system. In this manner, rearview vision system 12may be a compass vision system or an information vision system.

In the embodiment of rearview vision system 12 having a dynamicallyadjusted value of distance Q, the spacing between boundaries 50 and 52will dynamically adjust in sequence with the adjustment of distance Q.Thus, as overlap zones 32, 34 move further away from the vehicle; forexample, in response to an increase in speed of the vehicle, boundarylines 50 and 52 will move closer together and vice versa. In thismanner, composite image 42 is dynamic, having image portions ofdynamically adaptive sizes.

Display 20 is of a size to be as natural as possible to the driver. Thisis a function of the size of the display and the distance between thedisplay and the driver. Preferably, the displayed image simulates animage reflected by a rearview mirror. As such, the size of display 20 isapproximately the combined areas of the three rearview mirrors (oneinterior mirror and two exterior mirrors) conventionally used withvehicles. As best seen by reference to FIG. 2, display 20 is preferablypositioned within the driver's physiological field of view withoutobstructing the view through the windshield. It is known that thedriver's field of view, with the head and eyes fixed forward, extendsfurther in a downward direction than in an upward direction. Display 20could be located above the vertical view through the windshield whereinthe display may be observed at the upward portion of the driver's fieldof view. However, the position for the display illustrated in FIG. 2 ispreferred wherein the display is within the lower portion of thedriver's field of view.

Display 20, in the illustrated embodiment, is a flat panel display, suchas a back-lit liquid crystal display, a plasma display, a field emissiondisplay, or a cathode ray tube. However, the synthesized image could bedisplayed using other display techniques such as to provide a projectedor virtual image. One such virtual display is a heads-up display. Thedisplay may be mounted/attached to the dashboard, facia or header, or tothe windshield at a position conventionally occupied by an interiorrearview mirror.

Although various camera devices may be utilized for image capturedevices 14, 16, an electro-optic, pixelated imaging array, located inthe focal plane of an optical system, is preferred. Such imaging arrayallows the number of pixels to be selected to meet the requirements ofrearview vision system 12. The pixel requirements are related to theimaging aspect ratio of the respective image capture devices, which, inturn, are a function of the ratio of the vertical-to-horizontal field ofview of the devices, as is well known in the art. In the illustratedembodiment, the imaging aspect ratio of side image capture devices 14 is2:1 and the image aspect ratio of central image capture device 16 isvariable down to 0.1:1. Such aspect ratio will produce images which willnot typically match that of commercially available displays. Acommercially available display may be used, however, by leaving ahorizontal band of the display for displaying alpha-numeric data, suchas portions of an instrument cluster, compass display, or the like, asillustrated in FIG. 3.

In the illustrated embodiment, image capture devices 14, 16 are CMOSimaging arrays of the type manufactured by VLSI Vision Ltd. ofEdinburgh, Scotland, which are described in more detail in co-pendingU.S. patent application Ser. No. 08/023,918, filed Feb. 26, 1993, byKenneth Schofield and Mark Larson for an AUTOMATIC REARVIEW MIRRORSYSTEM USING A PHOTOSENSOR ARRAY, now U.S. Pat. No. 5,550,677, thedisclosure of which is hereby incorporated herein by reference. However,other pixelated focal plane image-array devices, which are sensitive tovisible or invisible electromagnetic radiation, could be used. Thedevices could be sensitive to either color or monochromatic visibleradiation or near or far infrared radiation of the type used innight-vision systems. Each image capture device could be a combinationof different types of devices, such as one sensitive to visibleradiation combined with one sensitive to infrared radiation. Examples ofother devices known in the art include charge couple devices and thelike.

As disclosed in U.S. patent application Ser. No. 08/023,918, filed Feb.26, 1993, by Kenneth Schofield and Mark Larson for an AUTOMATIC REARVIEWMIRROR SYSTEM USING A PHOTOSENSOR ARRAY, now U.S. Pat. No. 5,550,677,incorporated above, the logic and control circuit comprises a logiccircuit, a clock, a random-access-memory (RAM), or other appropriatememory, and a digital-to-analog converter. The logic circuit ispreferably a dedicated configuration of digital logic elementsconstructed on the same semiconductor substrate as the photosensorarray. Alternatively, the logic circuit may also be a microprocessorcomprising a central processing unit (CPU) and a read-only-memory (ROM).The logic circuit may also be implemented using gate array technology orany other appropriate hardwired logic circuit technology. The logiccircuit interfaces with the clock, provides array control signals to thephotosensor array, manages data flow to and from the RAM and convertersand, and performs all computations for determining, for example, adigital mirror control signal for causing the variable reflectancemirror element to assume a desired reflectance level. Theanalog-to-digital converter converts the analog photosensor elementsignals to the digital photosensor element signals processed by thelogic circuit. It has been found that an eight-bit analog-to-digitalconverter provides adequate data resolution for controlling the mirrors.Preferably, the analog-to-digital converter is constructed on the samesemiconductor substrate as the photosensor array. The digitalphotosensor element signals output to the logic and control circuit aregenerally stored in the RAM for processing. The values of the digitalphotosensor element signals for the photosensor array arecorrespondingly stored in an array in the RAM. The logic circuitprocesses the values of each of the digital photosensor element signalsto determine, for example, an instantaneous or substantially real-timebackground light signal for a time period t and at least one peak lightsignal. The logic circuit uses these signals, which may also betemporarily stored in the RAM, to determine a digital control signal to,for example, cause at least one mirror or mirror segment to assume adesired reflectance level. The digital mirror control signal may beoutput to the digital-to-analog converter, which may output acorresponding analog mirror control signal to a mirror drive circuit.Alternatively, the digital-to-analog converter need not be used if thelogic circuit generates a pulse-width-modulated (PWM) mirror controlsignal to control the mirror drive circuit. In one embodiment of theinvention, the logic circuit determines the background light signal bycalculating the average value of the photosensor element signals,previously stored in RAM, for the photosensor elements in a lowest rowor rows of the photosensor array corresponding to an area below the rearwindow. This means that the background light signal may be determinedfrom photosensor element signals generated by the photosensor elementslocated in one row of the photosensor matrix array. The logic circuitmay then output to the RAM for later processing. The logic circuit mayalso determine a background light level by calculating an average valueof all of the photosensor element signals in the entire photosensorarray. More generally, the background light signal for the rearwardscene may be determined by calculating the average value of X percent ofthe lowest photosensor element signal values in the RAM array, where Xis preferably 75, but typically may be in the range of 5 to 100.

The photosensor array is typically accessed in scan-line format, withthe array being read as consecutive rows, and within each row asconsecutive columns or pixels. Each photosensor element is connected toa common word-line. To access the photosensor array, a vertical shiftregister generates word-line signals for each word-line to enable eachrow of photosensor elements. Each column of photosensor elements isconnected to a bit-line which is connected to a charge-to-voltageamplifier. As each word-line is accessed, a horizontal shift registeruses a line to output the bit-line signals on consecutive bit-lines toan output line connected to the logic and control circuit. A voltageamplifier is used to amplify the resulting analog photosensor elementsignals. The analog photosensor element signals are then output on lineto the analog-to-digital converter and converted to digital photosensorelement signals.

Preferably, image capture devices 14 and 16 are all mounted at the samevertical height on vehicle 10, although compromise may be required inorder to accommodate styling features of the vehicle. The horizontal aimof image capture devices 14 and 16 is preferably horizontal. However,the portion of the image displayed is preferably biased toward thedownward portion of the captured image because significantly less usefulinformation is obtained above the horizontal position of the imagecapture devices.

Each image-capturing device 14, 16 is controlled by appropriatesupporting electronics (not shown) located in the vicinity of theimaging array such that, when operating power is supplied, either ananalog or a digital data stream is generated on an output signal linesupplied to image processor 18. The support electronics may be providedpartially on the image chip and partially on associated electronicdevices. For each exposure period, a value indicative of the quantity oflight incident on each pixel of the imaging array during the exposureperiod is sequentially outputted in a predetermined sequence, typicallyrow-by-row. The sequence may conform to video signal standards whichsupport a direct view such that, when a scene is viewed by animage-capturing device, the image presented on a display representsdirectly the scene viewed by the image-capturing devices. However, whenlooking forward and observing a displayed image of a rearward scene, thedriver will interpret the image as if it were a reflection of the sceneas viewed through a mirror. Objects to the left and rearward of thevehicle, as viewed by the rearward-looking camera, are presented on theleft-hand side of the display and vice versa. If this reversal iseffected in image processor 18, it may be by the use of a data storagedevice, or buffer, capable of storing all of the pixel values from oneexposure period. The data is read out of the data storage device in areversed row sequence. Alternatively, the imaging array electronicscould be constructed to provide the above-described reversal at theimage-capturing device or at the display.

Data transmission between image capture devices 14, 16 and imageprocessor 18 and/or between image processor 18 and display 20 may be byelectrically conductive leads or fiber-optic cable. It is possible, forparticular applications, to eliminate image processor 18 and directdrive display 20 from image capture devices 14, 16 at the pixel level.

The data streams from image-capturing devices 14, 16 are combined inimage processor 18 and directly mapped to the pixel array of display 20.This process is repeated preferably at a rate of at least 30 times persecond in order to present an essentially real time video image. Theimage captured by side image capture device 14 on the right side of thevehicle is presented in right image portion 46 and the image from sideimage capture device 14 on the left side of the vehicle is displayed onleft image portion 44. The image from center image capture device 16 isdisplayed on central image portion 48. The three image portions 44-48are presented in horizontal alignment and adjacent to each other.However, the composite image may be positioned at any desired verticalposition in the display 20. It is also possible to display imageportions 44-48 on separate image devices which are adjacent each other.

In vision system 12, side image capture devices 14 are positionedpreferably at a forward longitudinal position on vehicle 10 and centerimage capture device 16 is positioned at a rearward longitudinalposition on the vehicle. As best seen by reference to FIG. 7, thispositioning creates a difference in the vertical angle between each sideimage capture device 14 and center image capture device 16 with respectto a fixed location P₁ that is a distance D₁ behind the vehicle. Thisdifference in sensing angle will cause each side image capture device 14to image an object located at P₁ on a horizontal row of pixels that isdifferent from the horizontal row of pixels that center image capturedevice 16 will image the same object. If the image is below thehorizontal centerline of the image capture device, it will be imaged ona lower row of pixels by center image capture device 16 than the row ofpixels it will be imaged by the side image capture devices 14, asillustrated in FIG. 7. This mismatch between horizontal pixel rows ofthe captured image is furthermore a function of the distance of thecaptured image from the rear of the vehicle. This can be understood byreference to FIG. 11 which presents a chart 90 having a first column 92of pixel lines n1, measured from the array centerline, at which anobject will be imaged by side image capture device 14 and a secondcolumn 94 of pixel lines n2, measured from the array verticalcenterline, at which the same object will be imaged by center imagecapture device 16. The result is that an object, which is captured byboth side and center image capture devices 14, 16, will be verticallydisjointed at the boundary of the displayed image, if the object iscaptured by more than one image capture device. The amount ofdisjointment will be greater closer to the vehicle and less at furtherdistances. If the object is elongated in the horizontal direction, suchas earth's horizon, bridges, or cross-markings on highways, then theobject will appear to be either broken or crooked.

In order to provide uniform display of laterally elongated images, arearview vision system 12′ is provided having a central image portion48′ which is processed differently from the image display portions 44′and 46′ produced by the side image capture devices (FIG. 8). Centralimage portion 48′ is reduced vertically, or compressed, by removingspecified scan lines, or pixel rows, from the image captured by centerimage capture device 16 in a graduated fashion. The difference in thepixel line at which an object will be imaged by each of the side andcenter image capture devices is a function of the distance D of theobject from the rear of the vehicle, with a greater variation occurringat shorter distances and the variation reducing to zero for infinitedistances. Therefore, the compression of central image portion 48′ isnon-linear, with substantially no compression at the vertical center ofthe image and greater compression at greater distances above and belowthe vertical center point of the image. This is accomplished by removingspecific lines from the center display in a graduated fashion with agreater number of lines removed further from the vertical center of theimage. The removed lines may be merely discarded in order to verticallyreduce the image. Alternatively, the data contained in the removed linesmay be utilized to modify the value of adjacent pixels above and belowthe removed line in order to enhance the quality of the compressedimage. Averaging, median filtering, or other such known techniques mayalso be used.

Each of right image portion 46′ and left image portion 44′ includes anupper portion 64 which extends above the compressed upper portion of thecentral image portion 48′. In the illustrated embodiment, upper portions64 are deleted in order to present a uniform upper horizontal boundaryfor display 20′. In the illustrated embodiment, the mismatch between thelower horizontal boundary of central image portion 48′ and each of theleft and right image portions provides a dead space 66 which provides avisual prompt to the user of the approximate location of the rearwardcorners S of vehicle 10. This dead space 66 in the image displayed ondisplay 20′ approximates the footprint occupied by vehicle 10 whenviewed from point C. This is particularly useful because it provides avisual indication to the driver that a vehicle passing vehicle 10, asviewed in either left image portion 44′ or right image portion 46′, isat least partially adjacent vehicle 10 if the image of the approachingvehicle is partially adjacent to dead space 66.

In an alternative embodiment, the vertical compression technique may beapplied to only a lower vertical portion of central image portion 48′.In most driving situations, objects imaged by rearward-facing imagecapture devices above the horizon are at a long distance from thevehicle while those below the horizon get progressively closer to thevehicle in relation to the distance below the horizon in the displayedimage. Therefore, compression of the upper vertical portion of thecentral image portion may be eliminated without significant reduction inperformance.

Compression of the central image portion may also advantageously beprovided horizontally, as well as vertically. Spatial separation ofcenter image capture device 16 from side image capture devices 14 causessimilar distortion, as that described above, in the horizontaldirection. This effect is spherical in nature and would require a morecomplex corrective action, such as compressing the image based upon theremoval of pixels from an approximation to concentric circles centeredon the center of the imaging array, or other techniques which would beapparent to those skilled in the art.

A rearview vision system 12″ includes an image display 20″ having acompressed central image portion 48″ and left and right image portions44″ and 46″, respectively (FIG. 10). A border 50′ between left sideimage 44″ and central image 48″ includes a vertical central borderportion 50 a′, an upper border portion 50 b′, and a lower border portion50 c′. Upper border portion 50 b′ and lower border portion 50 c′ divergelaterally outwardly, vertically away from central portion 50 a′. Aborder 52′ between central image portion 48″ and right image portion 46″includes a central boundary portion 52 a′, an upper boundary portion 52b′, and a lower boundary portion 52 c′. Upper boundary portion 52 b′ andlower boundary portion 52 c′ diverge laterally outwardly vertically awayfrom central portion 52 a′. This creates an upper portion of centralimage portion 48″ and a lower portion of central image portion 48″ whichextend beyond the center portion thereof. This configuration is basedupon the realization that the surface of the road immediately behind thevehicle is captured by central image capture device 16. Likewise, thehorizontal plane above the vehicle, which is symmetrical with the roadsurface, is captured by the center image capture device. This may beseen by referring to point P in FIG. 10, which illustrate the pointswhere the effective radius 68 of the virtual image capture deviceintersects dead zones 30 and by referring to point S in FIG. 10 whichillustrates the corners or the rear of the vehicle (S).

The image displayed on display 20″ includes a dead space 66′ havingdiverging lateral sides 68 a, 68 b. Diverging sides 68 a and 68 b areconfigured in order to extend in the direction of travel of vehicle 10which is parallel to lane markings of a highway on which vehicle 10 istravelling. This further enhances the visual perception of the driver byproviding a visual clue of the location of images appearing on display20″ with respect to the vehicle 10. Side portions 68 a, 68 b, in theillustrated embodiment, are natural extensions of lower boundaryportions 50 c′ and 52 c′ and extend from point S on each respective sideof the vehicle to point R, which represents the intersection of thelower extent of the vertical field of view 40 of each side image capturedevice 14 with the pavement (FIG. 7).

Rearview vision systems 12′ and 12″ utilize a displayed synthesizedimage which takes into account the use of perspective in enhancing thedriver's understanding of what is occurring in the area surrounding thevehicle. The images produced on displays 20′ and 20″ effectively removethe vehicle bodywork and replace the bodywork with a vehicle footprintas would be viewed by virtual camera C. The image displayed on display20″ further includes perspective lines which further enhance the roll ofperspective in the driver's understanding of what is occurring.

In order to further enhance the driver's understanding of what isoccurring in the area surrounding the vehicle, a rearview vision system12′″includes a display 20′″ having image enhancements (FIG. 6). In theillustrative embodiment, such image enhancements include graphicoverlays 70 a, 70 b which are hash marks intended to illustrate to thedriver the anticipated path of movement of vehicle 10. In theillustrated embodiment, the anticipated vehicle motion is a function ofthe vehicle direction of travel as well as the rate of turn of thevehicle. The forward or rearward direction of vehicle travel isdetermined in response to the operator placing the gear selection device(not shown) in the reverse gear position. The degree of turn of thevehicle may be determined by monitoring the movement of the vehiclesteering system, monitoring the output of an electronic compass, ormonitoring the vehicle differential drive system. In the embodimentillustrated in FIG. 6, the configuration of graphic overlays 70 a, 70 bindicates that the vehicle is in reverse gear and that the wheels areturned in a manner that will cause the vehicle to travel toward thedriver's side of the vehicle. If the wheels were turned in the oppositedirection, graphic overlays 70 a, 70 b would curve clockwise toward theright as viewed in FIG. 6. If the vehicle's wheels were straight,graphic overlays 70 a, 70 b would be substantially straight converginglines. If the vehicle is not in reverse gear position, graphic overlays70 a, 70 b are not presented. Other types of graphic overlays of thedisplayed image are comprehended by the invention.

Horizontal grid markings on the display may be provided to indicatedistances behind the vehicle at particular markings. Such grid wouldallow the driver to judge the relative position of vehicles behind theequipped vehicle. In one embodiment, short horizontal lines aresuperimposed on the displayed image at regular rearward intervals inhorizontal positions which correspond to the boundaries of the lane inwhich the vehicle is travelling. In order to avoid confusion when thevehicle is travelling in a curved path, from a lack of correspondencebetween the graphic overlay and the road, a signal indicative of thevehicle's rate of turn may be taken into account when generating thegraphic overlay. In this manner, the distance indications may be movedlaterally, with reduced horizontal separation, to correspond to thepositions of the curved lane boundaries and vertically on the image tocompensate for the difference between distances along a straight andcurved path.

Another image enhancement is to alter the appearance of an object in aparticular zone surrounding the vehicle in order to provide anindication, such as a warning, to the driver. As an example, a vehiclethat is too close to the equipped vehicle for safe-lane change, may bedisplayed in a particular color, such as red, may flash, or otherwise bedistinguishable from other images on the display. Preferably, the speedof the equipped vehicle 10, which may be obtained from known speedtransducers, may be provided as an input to the rearview vision systemin order to cause such warning to be a function of the vehicle speedwhich, in turn, affects the safe separation distance of vehicles. Theoperation of the turn signal may also be used to activate suchhighlighting of other road users or to modify the scope of the imagedisplayed. In order to determine the distance of objects behind vehicle10, a separate distance-measuring system may be used. Such separatesystem may include radar, ultrasonic sensing, infrared detection, andother known distance-measuring systems. Alternatively, stereoscopicdistance-sensing capabilities of side image capture devices 14 may beutilized to determine the separation distance from trailing objectsutilizing known techniques.

Thus, it is seen that the image displayed on display 20-20′″ may bedifferent under different circumstances. Such different circumstancesmay relate to the vehicle's direction of travel, speed, rate of turn,separation from adjacent objects, and the like.

Various other forms of image processing may be utilized with rearviewvision system 12-12′″. Luminant and chrominant blending may be appliedto the images captured by image capture devices 14, 16 in order toproduce equality of the image data whereby the image portions appear asif they were produced by one image capture device. The dynamic range ofthe image capture devices may be extended in order to provide highquality images under all lighting conditions. Furthermore, individualpixel groups may be controlled in order to selectively compensate forbright or dark spots. For example, anti-blooming techniques may beapplied for bright spots. Multiple exposure techniques may be applied tohighlight dark areas. Image morphing and warping compensation techniquesmay additionally be applied. Resolution of the image capture devices anddisplay may be selected in order to provide sufficient image quality forthe particular application.

A heater may be applied to each image capture device in order to removedew and frost that may collect on the optics of the device. Although, inthe illustrative embodiment, the optical centerline of the cameracoincides with the field of view, particular applications may result inthe centerline of the camera pointing in a direction other than thecenterline of the field of view. Although, in the illustrativeembodiment, the image capture devices are fixed, it may be desirable toprovide selective adjustability to the image capture devices or opticalpaths in particular applications. This is particularly desirable whenthe system is used on articulated vehicles where automated andcoordinated camera aim may be utilized to maintain completeness of thesynthesized image.

When operating the vehicle in the reverse direction, it may be desirableto provide additional data concerning the area surrounding the immediaterear of the vehicle. This may be accomplished by utilizingnon-symmetrical optics for the center image capture device in order toprovide a wide angle view at a lower portion of the field of view.Alternatively, a wide angle optical system could be utilized with theelectronic system selectively correcting distortion of the capturedimage. Such system would provide a distortion-free image while obtainingmore data, particularly in the area surrounding the back of the vehicle.

The invention additionally comprehends the use of more than three imagecapture devices. In addition to side image capture devices positioned atthe front sides of the vehicle and a center image capture devicepositioned at the center rear of the vehicle, additional image capturedevices may be useful at the rear corners of the vehicle in order tofurther eliminate blind spots. It may additionally be desirable toprovide an additional center image capture device at a higher elevationin order to obtain data immediately behind the vehicle and thereby fillin the road surface detail immediately behind the vehicle. Suchadditional detail is particularly useful when operating the vehicle inthe reverse direction. Of course, each of the image capture devicescould be a combination of two or more image capture devices.

Although the present invention is illustrated as used in a rearviewvision system, it may find utility in other applications. For example,the invention may be useful for providing security surveillance in anarea where a building or other object obstructs the view of the areaunder surveillance. Additionally, the invention may find application innight-vision systems and the like. For example, the invention may beapplied to forward-facing night-vision systems, or other visionenhancement systems such as may be used in adverse weather oratmospheric conditions such as fog, applied to provide an enhanceddisplay of a synthesized image, which approximates a forward-facing viewfrom a single virtual camera located rearwardly of the driver, takingadvantage of the perspective features of the image.

Thus, it is seen that the present invention enhances the relationshipbetween the driver's primary view and the image presented on therearview vision system. This is accomplished in a manner which providesease of interpretation while avoiding confusion so that the driver doesnot have to concentrate or look closely at the image. In this manner,information presented on the display is naturally assimilated. This isaccomplished while reducing blind spots so that other vehicles orobjects of interest to the driver will likely be displayed to thedriver. Additionally, the use of perspective allows distances to be moreaccurately determined.

Changes and modifications in the specifically described embodiments canbe carried out without departing from the principles of the invention,which is intended to be limited only by the scope of the appendedclaims, as interpreted according to the principles of patent lawincluding the doctrine of equivalents.

The embodiments of the invention in which an exclusive property orprivilege is claimed are defined as follows:
 1. A driver assistancesystem suitable for use in a vehicle, said driver assistance systemcomprising: a first imager disposed at a left side of a vehicle equippedwith said driver assistance system, said first imager having a firstfield of view exterior of the equipped vehicle; said first imagercomprising a first two-dimensional CMOS array of photosensor elements; asecond imager disposed at a rear of the equipped vehicle and having asecond field of view exterior of the equipped vehicle; said secondimager comprising a second two-dimensional CMOS array of photosensorelements; a third imager disposed at a right side of the equippedvehicle and having a third field of view exterior of the equippedvehicle; said third imager comprising a third two-dimensional CMOS arrayof photosensor elements; wherein said first imager at said first side ofthe equipped vehicle is spatially separated from said second imager atsaid rear of the equipped vehicle and wherein said third imager at saidsecond side of the equipped vehicle is spatially separated from saidsecond imager at said rear of the equipped vehicle; wherein said firstfield of view overlaps said second field of view and wherein said thirdfield of view overlaps said second field of view; wherein a photosensorelement of said first two-dimensional CMOS array of photosensor elementsis individually accessible; wherein a photosensor element of said secondtwo-dimensional CMOS array of photosensor elements is individuallyaccessible; wherein a photosensor element of said third two-dimensionalCMOS array of photosensor elements is individually accessible; a displaysystem comprising a single display screen that is viewable by a driverof the equipped vehicle when the driver is normally operating theequipped vehicle; wherein said display screen is operable to display animage derived from image data captured by said first, second and thirdimagers, and wherein said display screen displays an image synthesizedfrom image data captured by each of said first, second and thirdimagers; and wherein the displayed image approximates a view from asingle virtual camera location.
 2. The driver assistance system of claim1, wherein the displayed image is substantially without duplicate imagesof objects.
 3. The driver assistance system of claim 1, wherein thedisplayed image approximates a substantially seamless view from thesingle virtual camera location.
 4. The driver assistance system of claim1, wherein the displayed image comprises a panoramic view rearward ofthe equipped vehicle.
 5. The driver assistance system of claim 1,wherein said first, second and third imagers are aimed alongnon-parallel axes.
 6. The driver assistance system of claim 1, wherein(i) said photosensor element of said first two-dimensional CMOS array ofphotosensor elements is individually accessible by at least one firstregister, (ii) said photosensor element of said second two-dimensionalCMOS array of photosensor elements is individually accessible by atleast one second register, and (iii) said photosensor element of saidthird two-dimensional CMOS array of photosensor elements is individuallyaccessible by at least one third register.
 7. The driver assistancesystem of claim 6, wherein (i) said at least one first registercomprises at least one of a horizontal shift register and a verticalshift register, (ii) said at least one second register comprises atleast one of a horizontal shift register and a vertical shift register,and (iii) said at least one third register comprises at least one of ahorizontal shift register and a vertical shift register.
 8. The driverassistance system of claim 1, wherein at least two of said first, secondand third imagers are at substantially the same height at the equippedvehicle.
 9. The driver assistance system of claim 1, wherein said firstand third imagers are at substantially the same height at the equippedvehicle.
 10. The driver assistance system of claim 1, wherein anelectronically generated overlay is superimposed on the image displayedby said single display screen to enhance the driver's understanding ofwhat is in the area adjacent to the equipped vehicle.
 11. The driverassistance system of claim 10, wherein said overlay enhances judgment bya driver of the equipped vehicle of the location of other vehicles andobjects external the equipped vehicle.
 12. The driver assistance systemof claim 11, wherein said overlay provides a visual cue to the driver ofthe equipped vehicle.
 13. The driver assistance system of claim 12,wherein said visual cue assists the driver to judge a distance to anobject rearward of the equipped vehicle.
 14. The driver assistancesystem of claim 1, wherein compression of at least one of said firstimage data, said second image data and said third image data in thesynthesized image reduces distortion in the displayed image.
 15. Thedriver assistance system of claim 1, wherein at least one of (a) saidfirst two-dimensional CMOS array of photosensor elements is formed on asemiconductor substrate, (b) said second two-dimensional CMOS array ofphotosensor elements is formed on a semiconductor substrate and (c) saidthird two-dimensional CMOS array of photosensor elements is formed on asemiconductor substrate.
 16. The driver assistance system of claim 1,wherein a first lens images light external the equipped vehicle ontosaid first two-dimensional CMOS array of photosensor elements and asecond lens images light external the equipped vehicle onto said secondtwo-dimensional CMOS array of photosensor elements and a third lensimages light external the equipped vehicle onto said thirdtwo-dimensional CMOS array of photosensor elements.
 17. The driverassistance system of claim 16, wherein at least one of said first lens,said second lens and said third lens comprises a plastic lens.
 18. Thedriver assistance system of claim 17, wherein each of each of said firstlens, said second lens and said third lens comprises a plastic lens. 19.The driver assistance system of claim 18, wherein each of said firstlens, said second lens and said third lens comprises a molded plasticlens.
 20. The driver assistance system of claim 1, wherein said displayscreen is disposed at or proximate an interior mirror assembly locationof the equipped vehicle.
 21. The driver assistance system of claim 1,wherein said display screen comprises a display screen selected from thegroup comprising (i) a plasma display screen and (ii) a field emissiondisplay screen.
 22. The driver assistance system of claim 1, whereinsaid display screen comprises a back lit liquid crystal display screen.23. The driver assistance system of claim 1, wherein said single virtualcamera location is external of the equipped vehicle.
 24. The driverassistance system of claim 23, wherein said single virtual cameralocation is forward of the equipped vehicle.